Performance evaluation of similarity measures for dense multimodal stereovision

Multi-modal imaging systems have recently been drawing attention in fields such as medical imaging, remote sensing and video surveillance systems. In such systems, estimating depth has become possible due to promising progress of multi-modal matching techniques. In this article, we perform a systematic performance evaluation of similarity measures frequently used in the literature for dense multi-modal stereo-vision. The evaluated measures include Mutual Information (MI), Sum of Squared Distances (SSD), Normalized Cross Correlation (NCC), Census Transform (CENSUS), Local Self Similarity (LSS) as well as descriptors adopted to multi-modal settings like SIFT, SURF, HOG, BRIEF and FREAK. We evaluate the measures over datasets we generated and compare the performances using the “Winner Takes All” (WTA) method. The generated datasets are (i) synthetically-modified four popular pairs from the Middlebury Stereo Dataset (namely, tsukuba, venus, cones and teddy), and (ii) our own multi-modal image pairs acquired using the infrared (IR) and the Electro-optical(EO) cameras of a Kinect device. The results show that MI and HOG provides promising results for multi-modal imagery and FREAK, SURF, SIFT and LSS can be considered as alternatives depending on the multi-modality level and the computational complexity requirements of the intended application.